<!doctype html>
<html lang="zh_cn" itemscope itemtype="http://schema.org/Person">
<head>
            <meta charset="utf-8">
        <!-- Site Meta Data -->
        <title>查看哪些进程在大量读写磁盘－I/O很高问题排查</title>
        <meta name="viewport" content="width=device-width, initial-scale=1">
        <meta name="description" content="技术博客,涉及Java/PHP/Python/Javascript等,聊聊程序,聊聊生活,聊聊事实,聊聊育儿">
        <meta name="keywords" content="编码经验,技术分享,生活积累,实事评说">
        <meta name="author" content="布丁缘">

        <link rel="shortcut icon" href="">

        <link href='https://fonts.googleapis.com/css?family=Open+Sans:400,600,700' rel='stylesheet' type='text/css'>
        <!-- Style Meta Data -->
        <link rel="stylesheet" href="https://www.ddkiss.com/theme/css/style.css" type="text/css"/>
        <link rel="stylesheet" href="https://www.ddkiss.com/theme/css/pygments.css" type="text/css"/>

        <!-- Feed Meta Data -->
            <link href="https://www.ddkiss.com/feeds/all.atom.xml" type="application/atom+xml" rel="alternate"
                  title="一个程序员的简单生活 ATOM Feed"/>


    <meta name="keywords" content="">
    <link rel="stylesheet" href="//dn-coding-net-public-file.qbox.me/Coding-Comments/v0.1.0/default.css">
</head>

<body>
<!-- Sidebar -->
<aside>
    <center><h1><a href="/" style="color:#fff"><img id="avatar" src="/images/avatar.jpg"></a></h1></center>
        <p>一个程序员的简单生活</p>
    <br>
    <nav class="nav">
        <ul class="list-bare">
                <li><a class="nav__link" href="https://www.ddkiss.com/category/chang-yong-ji-qiao.html">常用技巧</a></li>
                <li><a class="nav__link" href="https://www.ddkiss.com/category/kai-fa-huan-jing.html">开发环境</a></li>
                <li><a class="nav__link" href="https://www.ddkiss.com/category/sheng-huo-dian-di.html">生活点滴</a></li>

                <li><a class="nav__link" href="/pages/books.html">书单</a></li>
                <li><a class="nav__link" href="/pages/movies.html">影单</a></li>
                <li><a class="nav__link" href="/pages/downloads.html">下载</a></li>
                <li><a class="nav__link" href="/pages/about.html">关于我</a></li>


        </ul>
    </nav>


    <form>
      <input type="text"  id="bdcsMain"/>
    </form>

</aside>

<!-- Content -->
<article>
  <main>
    <nav>
      <a href="/">首页</a>
      <a href="/archives.html">归档</a>
      <a href="/categories.html">分类</a>
      <a href="/tags.html">标签</a>
      <a href="/pages/about.html">关于我</a>
    </nav>
  </main>
    <section id="content">
        <article>
            <h2 class="post_title post_detail"><a href="https://www.ddkiss.com/archives/68.html" rel="bookmark"
                                                  title="Permalink to 查看哪些进程在大量读写磁盘－I/O很高问题排查">查看哪些进程在大量读写磁盘－I/O很高问题排查</a></h2>

            <div class="post_list">
                <span>作者：</span><a href="https://www.ddkiss.com/author/bu-ding-yuan.html">布丁缘</a>
                <span class="post_category">分类：<a href="https://www.ddkiss.com/category/kai-fa-huan-jing.html" rel="bookmark"
                                               title="Permalink to 开发环境">开发环境</a></span>
                <span class="post_date">  时间：2017-10-20 10:29:00</span>

            </div>
            <div class="entry-content blog-post">
                <p>Linux 有很多运维诊断工具，有些用起来很简单，有些功能很强大用起来就有些麻烦。比如I/O等待问题，原因可能有很多种，也很难用某一种工具就能定位。特别是如何找到到底是哪个进程读写了哪个文件引起的？</p>
<p>最近，从监控上看总是在上午10点左右I/O负载突然就升起来了，连远程SSH都卡住。因为ECS服务器内存有限，我启用了SWAP，内存不足时不至于会崩溃，系统性能肯定有所下降。</p>
<p>说明：在Centos7中，<code>top ps</code>等是系统自带的工具，<code>iostat iotop lsof</code>等都是需要自行安装的。</p>
<h2>确实系统负载上升是否由于I/O异常导致</h2>
<p>最简单的是用<code>top</code>命令，比如</p>
<div class="highlight"><pre><span></span><span class="n">Tasks</span><span class="o">:</span> <span class="mi">102</span> <span class="n">total</span><span class="o">,</span>   <span class="mi">1</span> <span class="n">running</span><span class="o">,</span> <span class="mi">101</span> <span class="n">sleeping</span><span class="o">,</span>   <span class="mi">0</span> <span class="n">stopped</span><span class="o">,</span>   <span class="mi">0</span> <span class="n">zombie</span>
<span class="o">%</span><span class="n">Cpu</span><span class="o">(</span><span class="n">s</span><span class="o">):</span>  <span class="mf">0.7</span> <span class="n">us</span><span class="o">,</span>  <span class="mf">0.2</span> <span class="n">sy</span><span class="o">,</span>  <span class="mf">0.0</span> <span class="n">ni</span><span class="o">,</span> <span class="mf">0.2</span> <span class="n">id</span><span class="o">,</span>  <span class="mf">98.9</span> <span class="n">wa</span><span class="o">,</span>  <span class="mf">0.0</span> <span class="n">hi</span><span class="o">,</span>  <span class="mf">0.0</span> <span class="n">si</span><span class="o">,</span>  <span class="mf">0.0</span> <span class="n">st</span>
<span class="n">KiB</span> <span class="n">Mem</span> <span class="o">:</span>  <span class="mi">1016396</span> <span class="n">total</span><span class="o">,</span>    <span class="mi">62276</span> <span class="n">free</span><span class="o">,</span>   <span class="mi">891072</span> <span class="n">used</span><span class="o">,</span>    <span class="mi">63048</span> <span class="n">buff</span><span class="o">/</span><span class="n">cache</span>
<span class="n">KiB</span> <span class="n">Swap</span><span class="o">:</span>  <span class="mi">1048572</span> <span class="n">total</span><span class="o">,</span>   <span class="mi">804428</span> <span class="n">free</span><span class="o">,</span>   <span class="mi">244144</span> <span class="n">used</span><span class="o">.</span>     <span class="mi">8100</span> <span class="n">avail</span> <span class="n">Mem</span>
</pre></div>


<p>其中<code>wa</code>也就是<code>iowait</code>参数所表示的，有多少比例的CPU在等待I/O</p>
<h2>找到具体读写的是哪个磁盘</h2>
<p>最简单的是<code>iostat</code>命令，比如</p>
<div class="highlight"><pre><span></span> $ iostat -x 2 5
 avg-cpu: %user %nice %system %iowait %steal %idle
  3.66 0.00 47.64 48.69 0.00 0.00

 Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
 sda 44.50 39.27 117.28 29.32 11220.94 13126.70 332.17 65.77 462.79 9.80 2274.71 7.60 111.41
 dm-0 0.00 0.00 83.25 9.95 10515.18 4295.29 317.84 57.01 648.54 16.73 5935.79 11.48 107.02
 dm-1 0.00 0.00 57.07 40.84 228.27 163.35 8.00 93.84 979.61 13.94 2329.08 10.93 107.02
</pre></div>


<p>其中<code>-x</code>知指明输出更加详细的信息。<code>2 5</code>表示间隔2秒统计一次总共输出5次。第一行结果是自系统启动以来的统计值，通常排查突发的I/O异常时可忽略。重点看<code>%util</code>列，表示进程使用I/O的比例。</p>
<h2>找出引起高I/O的进程</h2>
<p>最简单的是<code>iotop</code>命令，比如</p>
<div class="highlight"><pre><span></span>Total DISK READ: 404.95 K/s | Total DISK WRITE: 9.91 M/s
  TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO&gt;    COMMAND                                                                                               
 3153 be/3 root        0.00 B/s    3.78 K/s  0.00 %  6.23 % [jbd2/sda5-8]
10287 be/4 mysql     253.57 K/s 1858.24 K/s  0.00 %  3.73 % mysqld --defaults-file=/etc/my.cnf --basedir=/opt/my~--pid-file
</pre></div>


<p><code>iotop</code>命令确实是一个很好用的工具，但是很多Linux发行版中默认并没有安装。如果系统里没有这个命令，又不想安装时，怎么办？有没有更简单的方式？</p>
<h2>能够实现<code>iotop</code>类型功能的<code>ps</code>命令</h2>
<p>首先看下进程状态编码</p>
<div class="highlight"><pre><span></span>PROCESS STATE CODES
 D uninterruptible sleep (usually IO)
 R running or runnable (on run queue)
 S interruptible sleep (waiting for an event to complete)
 T stopped, either by a job control signal or because it is being traced.
 W paging (not valid since the 2.6.xx kernel)
 X dead (should never be seen)
 Z defunct (&quot;zombie&quot;) process, terminated but not reaped by its parent.
</pre></div>


<p>注意到，如果进程正在等待I/O，进行的状态码经常为<code>D</code>。通过这个信息，我们可以大概估算出哪些进程在I/O等待中，比如</p>
<div class="highlight"><pre><span></span>[root@myos ~]# for x in `seq 1 1 10`; do ps -eo state,pid,cmd | grep &quot;^D&quot;; echo &quot;----&quot;; sleep 5; done
----
----
----
D 27456 php-fpm: pool www
----
----
----
----
----
----
----
</pre></div>


<p>这里就发现进行 27456　可能有问题。但等待I/O也不一定说明它在疯狂读写磁盘。为了确定它确实读写很多，可以用<code>/proc</code>目录下的信息来定位，比如</p>
<div class="highlight"><pre><span></span>[root@myos ~]# cat /proc/27456/io
rchar: 42857442
wchar: 35657666
syscr: 44932
syscw: 9744
read_bytes: 1530859520
write_bytes: 13824000
cancelled_write_bytes: 1249280
</pre></div>


<p>需要注意的是，不同的线程参考值不一样。这个得具体问题具体分析。</p>
<h2>参考</h2>
<p><a href="http://bencane.com/2012/08/06/troubleshooting-high-io-wait-in-linux/">英文原文</a></p>
            </div>
            <div class="post_list">
              <div><span>Tags : </span>
              </div>
            </div>
        </article>
        <div id="container"></div>
        <script type="text/javascript" src="//dn-coding-net-public-file.qbox.me/Coding-Comments/v0.1.0/gitment.min.js"></script>
        <script>
            var gitment = new Gitment({
              owner: 'whusl',
              repo: 'BlogComments',
              oauth: {
                client_id: '621866266817529fba46681653017809',
                client_secret: '14188411740b12ae52159cee9b586bf85cd54125',
              },
            })
            document.getElementById('container').appendChild(gitment.render())
          </script>
    </section>
</article>

<!-- Footer -->
    <footer>
        <p> &copy;2017-2020&nbsp;<a href="http://www.miitbeian.gov.cn/" target="_blank">鄂ICP备17020200号</a>
          Blog powered by <a href="http://getpelican.com/">Pelican</a>
        </p>
    </footer>

    <!-- Analytics -->
    <script>
      var _hmt = _hmt || [];
      (function() {
        var hm = document.createElement("script");
        hm.src = "https://hm.baidu.com/hm.js?88c55edaf311dbacac56a16316b04c8b";
        var s = document.getElementsByTagName("script")[0];
        s.parentNode.insertBefore(hm, s);
      })();
    </script>

<script type="text/javascript">(function(){document.write(unescape('%3Cdiv id="bdcs"%3E%3C/div%3E'));var bdcs = document.createElement('script');bdcs.type = 'text/javascript';bdcs.async = true;bdcs.src = 'http://znsv.baidu.com/customer_search/api/js?sid=14490611060029767912' + '&plate_url=' + encodeURIComponent(window.location.href) + '&t=' + Math.ceil(new Date()/3600000);var s = document.getElementsByTagName('script')[0];s.parentNode.insertBefore(bdcs, s);})();</script>

</body>
</html>